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Measurement is prescriptive in that the model 
required to produce the numbers from the phenomena is known in 
advance* Statistical theory is descriptive because the model required 
to explain the numbers is inferred from the numbers and any other 
information the researcher can bring to bear* There are well-known 
families of measurement models and statistical models; combining them 
poses a number of problems, as is illustrated through an analysis of 
the relationship between literacy and the last grade level completed 
for 17,650 respondents to the National Adult Literacy Survey* One 
approach is to use measurement and then description by summarizing 
the literacy of adults by their last grade levels* A Rasch model 
approach can be used* Another approach, that of explanatory 
measurement, is to attempt to discover how much can be explained by 
last grade level and how much by demography* The two analyses are 
based on the same data but address different questions* If the sample 
used for the summary analysis adequately represents the country’s 
population, then the results summarize the population's literacy as 
well* The explanatory analysis enables prediction of expected reading 
comprehension levels* The choice of analysis depends on whether the 
researcher wants a picture of the situation or insight into its 
formation* (Contains four figures and five references*) (SLD) 



ED 398 277 

AUTHOR 
TITLE 
PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



ABSTRACT 



'k k Vc Vc y c k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k k kkkk k k k k k k 

Reproductions supplied by EDRS are the best that can be made 
* from the original document* 

yc^ycVt kkkkkkkk k k k k kkk kk k k kkkkkk kkk kkkk kk kkkkk kkkkkk k k k kkkkkkkkkkkkkk k k kk k 




ED 398 211 



II Q department OF EDUCATION 
OH^olEducational Research 

Educational resources information 
/ CENTER (ERIC) 

cyrhis document has been 
^ received from the person or organization 

originating it. 

O Minor changes have been made to 
improve reproduction quality. 

• Points of view or opinions 

document do not necessarily represent 
olficial OERl position or policy 



Descriptive or Explanatory Measurement 



PERMISSION TO REPRODUCE AND 

disseminate this material 
HAS BEEN GRANTED BY 

ya/jA) H ' , ^ 



> 

a*"* 



E-mail 



John M. Linacre 

MESA Psychometric Laboratory, Univenity of Chicag^^^^ 
MESA@uchicago.edu, Tel. (312) 702-1596 FAX (312) 702-0248 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



With summarizing such numbers into a few par^ete ■ advance Statistical theory is 

(regression models). Can rhese de combined 
into a one-step procedure? ^ 

OeorgRa.b(ormuimed^mode,.am_^ 

makes items hard (Smith et al. 1992). 

Since the Rasch model is transpamnt 

persons also could be ‘decomp^rf • as a general change in item 

(or change, in which a general chmge P rarameters howeve' can be easily accomplished, 
difficulty Direct decomposition of person ability parameters, nowe e., y 

NevertlSless problems in selection, interpretation and commumcation arise. 

Methods of Analysis 

-h adult is measured m^ «h item 

calteatS; Each adult oLns a measure and a standard error (which can be misfit inflated). 

These measures are summarized by demographic factor (unadjusted for other factors). 

This is the typical report displayed in newspapers. 

Here problems of interpretation and communication arise. 

Each adult is “decomposed” demographically according to gender, ethnicity, 

Thl iSuces the prablem of decomposition selection. Many demographic variables are highly 

correlated. 

The sizes of the demographic effects (and their standard errors) are estimated using regression. 
;^etfc^ta unweighted, S.E.-weighted (i.e„ by effea stas), or itifonnation-weighmd. 
This is the typical report displayed in research findings. 

In summary, 

response data = individual measures + stochasticity + miSfit 
inchvidual measures = demographic effects + unexplained variance 
unexplained variance = individual variance + measurement error 
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Explanatory measurement: 

Each adu^lt is “decomposed” demographically according to gender, ethnicity, native language etc 
i.e., each adult is treated as a random representative of the individual’s demography 

A many-facet R^ch analysis is performed in which each demographic effect is measured. 

This introduce the problem of identifying the frame of reference. Though trivial mathematically 
It presents difficulties m mterpretation and communication. 

The sizes of the demographic effects (and their standard errors) are estimated. 

These reports ore not yet in public circulation. 

In summary, response data = demographic effects + stochasticity + misfit 

']^e logit sizes of the demographic effects estimated by the two methods will differ Since the 
stochasticity in method 2 comprises the stochasticity in method 1 plus the within-demographic-effcet 

person variance, then method 2 will appear to be less discriminating. The logit difference befweeftwo 
Items m method 2 will be smaller than m method 1 . ainerence between two 

Heteroscedascity of mdivi^uals within demographic cells will increase misfit and so distort the 

demographic effect mean is obtained chSty; no 
withm-demographic-effect sample variance is obtained. On the other hand, BayLian 

g3p^Sh“:^:^^^ 

This study raises philosophical problems about “what we want to measure?” analogous to those raised 

^d Rasch measures in answer to the question “how we want to measure*^” 
Most audiences have been familiarized to raw scores, and feel uncomfortable with measures (despite their 
obvious adv^tages). Similarly, most audiences have been familiarized with descS^ 

Ze7 ^ I Z Are explanatory measurement reports (2c) an ideal to be 

amed at, or a nusleading distraction? The audience is encouraged to consider this question by comparing 
plots depicting different answers to the question “what size is the demographic effect?” ^ ^ 

The NALS Literacy Survey 

^ investigation of the relationship between literacy and last grade level completed 24 944 
subjects were mterviewed for the 1992 National Adult Literacy Survey. Of these, 17,650 subjects have 

test responses and demographic information. There were 184 literacy iteiis on the 
urvey. Of these, 165 items provide interpretable literacy information. 

Measurement then Description 

One approach to the relationship between last grade level completed and adult literacy is to summarize 

the iteracy of adults by last grade level. A Rasch analysis produces literacy measures^or 17 650 adults 
on 165 Items. The measurement model is y ror i / ,odu aauits 



log 



P.. 






measures are then summarized by grade level. This description is a snap-shot of historv 

subjected to the usual regression analysis. Since each measure has a standard error a weighted 
egression could be performed. Weighting by inverse standard error ( 1/S.E.), rather than by inforit^tion 
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(1/S.E.2), is likely to be more useful (Wright 1994). 




Figure 1. Reading Measures Summarized by Grade Level 



Explanatory Measurement 

Adult literacy is affected by many demographic and other factors. In an attempt to discover how much 
is explained by last grade level and how much by demography, an alternative demographic model is 
constructed. Now each respondent is thought of as a random member of each relevant demographic 
group. Each demographic group is parameterized to manifest one fixed effect. The demographies are 
last grade completed (Gg), personal income (M J, first language (L,), ethnicity (EJ, age (A,) and sex (S^. 
The explanatory measurement model is: 
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The grade effect measures this model produces are shown in the plot in Figure 2. 

Why the differences? 

Though the trend in the two plots are similar, there are clear differences. Figure 3 plots the two sets of 
grade level literacy measures. The relationship is close to linear, but not an identity line. The logit range 
for the summary measures is twice that for the explanatory measures. There are two reasons for this. 



a) The length of the logit: 

The measures are reported in logits, a probabilistically defined unit of measurement. The relationship 
between logits and the substantive definition of the latent variable depends on the amount of stochasticity 
in r’le data. The more noisy the data are, the more substantive advance along the latent variable there 
is per logit. The less noisy the data are, the larger (in logits) any given substantive difference appears. 
In perfect Guttman data, every respondent is infinitely far from every other respondent with a different 
raw score on the same items. 



In the NALS example, the measurement model with 17,650 respondent parameters explains much more 
of the variance in the data, than the explanatory model with 54 demographic parameters. The 
unexplained variance, i.e., that within demographic groups, becomes part of the unexplained noise, 
inflating the measurement error. This larger measurement error reduces the logit range of the reported 
measures. 
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Figxire 2. Explanatory Grade Measures. 



NALS Sample by Grade 




Figure 3. Comparison of Grade Level Measures. 



The size of the logit change can be discovered by comparing the standard deviations of the item 
calibrations for the two analyses. In Figure 4, the standard deviation of the calibrations for the individual 
person measures is 1.2 times that for the explanatory model. This comparison of item calibrations 
equates the two alternative analyses by means of a part of the analysis that we can consider invariant. 

b) Collinearity of explanatory variables: 

Higher reading comprehension would be expected of those respondents with more education, but also of 
those respondents with higher income. A likely contributing factors to higher income are both more 
education and more reading comprehension. The explanatory model partitions the reading comprehension 
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NALS: 165 Items 




Figure 4. Coirijparison of item calibratioas. 



effect of the demographic variables for this sample. According to Figure 3, additional education by itself 
(according to the explanatory run) accounts for about half the increase in literacy achieved by those with 
more education. After adjusting for the change in logit size, 63% of the increase is due to additional 
education. 



Descriptive or Explanatory Measurement? 

These two analyses, though based on the same data, answer different questions. The summary analysis 
reports the reading comprehension associated with each last grade level according to this sample. If this 
sample adequately represents the US population, then these results also summarize the population. The 
explanatory analysis enables us to predict, for any people similar to those sampled, from their 
demographic characteristics what are their expected reading comprehension levels. The concave form 
of Figure 3 indicates that the summary statistics underemphasize the close relationship between reading 
comprehension and successful completion of higher education. The choice of analysis depends on 
whether the researcher desires a snapshot of the current situation or insight into its formation. 
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